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ABSTRACT 

Three types of evaluation used in program evaluat ^n 
and the type of instrument or measure used are discussed. The three 
types of evaluation are: comparison group evaluation, analysis of 
learner outcome or product objectives, and process evaluation. 
Measurements employed are either direct or indirect. Most measurement 
in education is usually indirect, by which the subject may be 
measured while he either actively or passively participates in the 
measurement act. When it is determined that a test will be employed 
as a means of indirectly sampling some behavior, two basic kinds of 
tests are employed — the standardized norm referenced test and a 
criterion referenced test. Once the evaluation type and 
instrumentation have been defined, there is the question of the 
consequences of evaluation. The object of evaluation is to provide 
decision- making information about programs. Steps in program 
evaluation are: (1) consideration of how objectives will be evaluated 
should begin prior to program development; (2) following consultation 
with evaluation personnel, the planning group should make decisions 
on what measurement instrument, performance objective formats, and 
c--iterion levels represent fair and reasonable assessment of learner 
outcome, product objectives; (3) a system for recording and reporting 
the results of process objective evaluation should be established in 
the school; and (4) product objectives and comparison group 
evaluation should take place at the end of the program. (DB) 
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PROCEDURES FOR PROGRAM EVAUJATION 



Much of vAiat has been said in the previous three planning dociiments 
has direct relationship to procedures for program evaluation. However, 
for the purpose of understanding the role of evaluation in planning it 
is necessary to understand only three types of evaluation. 

The first type, called comparison grcnip evaluation, is done apart 
from any performance objectives established for the program. Ccmiparison 
evaluation simply means that two or more programs trsring to accoo^lish 
similar ends are compared to see idiich one is most successful. In order 
to conduct this Icind of evaluation, it is necessary to design the program 
implementation In such a way that there is random assignment of students 
to various competing programs. 

It is interesting to note that comparison group evaluation can take 
place without the specification of performance objectives for a program. 
Uhfortunately the perfonnance objectives are too often thou^^it of as 
merely an extension of evaluation, as opposed to their flindamental ijiipor- 
tance in program development. 

The second type of program evaluation is the analysis of learner 
outcome or product objectives. Like the coinparison group evaluation, the 
evaluation of learner outcome objectives i^eveals infomation on the 
overall success or failure of the program. However, the evaluation of 
these learner outcome objectives in isolation gilres an absolute rather 
than a relative measure of success. For instance, two reading programs 



that attained all of their learner outcome objectives might be ranked 
equally successflil on this basis, but differ markedly if they could be 
compared one to another in a comparison group evaluation* 

A third type of evaluation is related to the process objectives 
that are used to admjj^Lster the program. I^ike the comparison evalua- 
tion and the px*oduct objectives evaluation , the process evaluation is 
related to the means of the program rather than the ends. To be of value , 
the process objectives are evaluated during the program's operaticm^ 
Based on the outcome of process objective evaluation, projects can be 
modified in ^midstream^ in order to increase the probability of attain- 
ing outcoone objectives. 

The foxmat for product and process objectives is to a degree a 

function of the type of measurement enployed. Aside from objectives that 

are sometimes written in terms of a comparison group evaluation, most 

other kinds of perforaance objectives are written to fit one or more of 

the following evaluation fomatss Examples 

(all product objectives) 

1. An historical cocuparison - that is, The average score on the ABC 
objectives are written to compare a test of reading coinprehension 
group's perfomance one year to the will be higher for 1972 Grade 
same group or a different group in 6 pupils than for 1971 Grade 6 
some previous year. pupils in Smith Elementazy 

School. 

2. Pre and post test coaQ>ari9ons - The 6th grade pupils in Smith 
that is, objectives are written Elementary School will average 
in terms of how much improvement one year gain In z*eadlng for the 
is shown during a one year period school year 1972-1973 m measured 
by the same group. by the ABC test of reading. 

3« Conqyaxlson to some norm that is, 80^ of the 6th grade students 
objectives are written with stand- at Sttith Elementary School will, 
ards of perfonnance related to the by the end of the 1972^1973 
results of norm referenced testing. school year, be reading at grade 

level as measured by the ABC test 

of reading. 



Content based, mastery level conn 85Jf of the 6th grade students 

parison - that is, structuring the at amith Elementary School will 

objectives so that the measurement demonstrate survival literacy by 

is based on some prior judgments of scoring at the correct level 

mastery level. This kind of or better on a mastery test of 

measurement is usually associated literacy* 
with criterion-*3?eferenced testing* 



Hopefully the above discussion has convinced the reader that these 
three kinds of eval\iation are all necesaeuty to maximizing 1^ information 
concerning program success or failure* Product objective and process ob«* 
jective evaluation can usually be conducted by program personnel* However, 
some assistance in specifying the performance objectives and creating record 
keeping documents for evaluation is usually needed* The contpeirison group 
evaluation, idiich need not be based upon the product or process objectives, 
is usxially conducted by an evaluation staff* It is important that the 
evaluation staff is contacted early enough prior to program implementation 
to explore with the planning group the possibilities of implementing the 
program in such a way as to provide for comparison group evaluation* 

The other important issue in evaluation is the type of measure or 
instrument eniployed* Measurements are either direct or indirect* In direct 
evaluation the behavior is cootpletely observable* For exaiq)le, the objective 
below can be eval\iated in a very direct manners 

All sixth grade students shall run the hundred 
yard dash in under twelve seconds* 



In this exaiq>ie the physical skill involved in running a hundred yard 
dash under twelve seconds is directly observable and directly measured* 



Most measurement in education usually is not of this direct type. 
Usually we gather a sanq^le of behavior that represents some kind of in- 
ternal activity. The typical achievement test 5n z^ading purports to 
indirectly sample a set of behaviors that are important to the total, 
internal act of reading. 

Two kinds of indirect measures can be distinguished by whether or 
not the subject being measured actively or passively participates in the 
measurement act. In the reading test example above, the student is re«- 
quired to demonstrate his reading knowledge by actively taking the test. 
However, if we were interested in measuring something like reading in* 
terest, good interest tests might not be readily available. Therefore, 
it may be necessary to devise other kinds of indirect measures to pick 
up samples of behavior that might have implications for reading interest* 
Pbr instance, voltmtary trips to the library, a record, of books checked 
out of the library, the amount of time spent in recreational reading, 
might all be observations frao which one could infer a reading interest 
level. This latter kind of unobtznisive, indirect measure is the kind 
usually employed in trying to assess student self concept, reading in- 
terests, attitude towards school, and other aspects of affective behavior. 

When it is determined that a test will be employed as a means of in- 
directly sanq;>ling some behavior, there are two basic kinds of tests 
employed. One is the standardized nozm referenced test with ^Aich most 
educators are quite familiar. The Iowa Test of Basic Skills is a 
standardized, nom referenced test. The other type is called a criterion 



referenced test, axid «Mle it quite often purports to measure the same 
thing as is measured by a nozm iTeferenced t^est, its construction! 
applicability, and resulting scoz^es are quite diff extent. There is not 
the time in this planning document to discuss the merits of each type of 
measurement except to suggest that the criterion referenced test is 
thought by some experts to be more applicable to planning. The reader is 
refezred to the Department of Educational Research and Program Assessment 
for detailed information on norm i^eferenced and cxlterion referenced 
testing. 

Th±B discussion of types 6t direct and indirect measuz^ement is 
ioqportant to the person or group charged vdth the responsibility of 
developing performance objectives. Clearly, product, perfozmance objec- 
tives need not be written solely based on some standardized test. The 
measurement of performance is limited only by the imagiziation of the 
objective writer and the limitations of measurement. The limitations 
of measurement are often ameliorated by combining measur^nent approaches. 
Each measurement approximates, and all approximations taken togc ^bar 
usually provide inore precision. 

Higner order conceptual activity often is difficult to define in 
perfozt&ance terms. However, this limitation applies to product objectives 
and not process objectives. Process objectives refer to how programs are 
implemented and are almost always amenable to description in perfozmance 
terms. Even certain learner outcomes that initially are thought to be 
irreducable to perfozmance description, can have some of their asi>ects 



described in performance terms. Pbr those learner outcomes that can't be 
measured, structured opinion-gathering instruments can be employed. 

Once the evaluation type and instrumentation have been defined, 
there still is the question of vhat are the consequences of evaluation. 
Here, unfortunately, the specter of personal blame too often clouds the 
evaluation. It is not "Uie purpose of program evaluation to ascribe fault 
to individuals or to reward others. Iftiile in some school systems, eval- 
uation results have been used in this fashion, it is not the intent or 
desire of the evaluation cooqponent of the Milwaukee Public School plan- 
ning process to be tised in this way. The object of evaluation is to 
provide decision-making infonnation about programs, not individuals. 
The consequence of not attaining desired student outcomes or not meeting 
process objectives, is to modify the program. 



steps in Program Evaluation: 



1. Consideration of how objectives will be evaluated should 
begin prior to program development. Working through the 
administrative specialisty the princii>al and his planning 
staff may wish to consult Department of Educational Research 
and Program Assessment personnel on questions of evaluation. 
If the planning group desires comparison group evaluationy it 
is particularly inqportant to contact evaluation personnel 
before program cooomitments have been made. 

2. Following consultation with evaluation personnel, -Uie planning 
group should make decisions on vAxat measurement instruments, 
performance objective formats, and criterion levels represent 
fair and reasonable assessment of learner outcome, product 
objectives. 

3. A system for recoixling and reporting the results of process 
objective evaluation should be established in the school. These 
evaluation results, received periodically throughout the year, 
can serve to modijEy the program while it is operating. 

4. Product objectives and comparison group evaluation should take 
place at the end of the program. The results of these evalua- 
tions coupled with the implementation assessment gained from 
process objective evaluation, will form the basis for a new 
needs assessment. 



